A Variance Controlled Stochastic Method with Biased Estimation for Faster Non-convex Optimization

This paper proposed a new technique Variance Controlled Stochastic Gradient (VCSG) to improve the performance of stochastic variance reduced gradient (SVRG) algorithm. To avoid over-reducing by SVRG, hyper-parameter \(\lambda \) is introduced in VCSG that able control SVRG. Theory shows optimization method can converge using an unbiased estimator, but practice, biased estimation allow more efficient convergence vicinity since approach computationally expensive. also has effect balancing trade-off between and estimations. Secondly, minimize number full calculations variance-bounded batch reduce required each iteration. For smooth non-convex functions, algorithm converges approximate first-order stationary point (i.e. \(\mathbb {E}\Vert \nabla {f}(x)\Vert ^{2}\le \epsilon \)) within \(\mathcal {O}(min\{1/\epsilon ^{3/2},n^{1/4}/\epsilon \})\) evaluations, which improves leading complexity gradient-based SCSG \((\mathcal ^{5/3},n^{2/3}/\epsilon [19]. It shown theoretically experimentally be deployed convergence.